Systems, methods and apparatus for characterizing stick-up height, position and orientation of a drill pipe

ABSTRACT

A processor is operably coupled to a time of flight (TOF) camera, a light detection and ranging (LIDAR) sensor, and an optical camera. The processor can receive a TOF signal representative of a first coordinate associated with a stick-up height of a tool joint of a pipe of a drill string during a tripping operation on a rig drill floor, and a pitch and a roll of the tool joint. The processor can receive a LIDAR signal representative of a second coordinate associated with the stick-up height, and the pitch of the tool joint. The processor can receive an optical camera signal representative of a third coordinate associated with the stick-up height of the tool joint and the roll of the tool joint. The processor can generate a pose estimate and an orientation estimate based on the signals.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/732,225 entitled “Systems, Methods and Apparatus for CharacterizingStick-Up Height, Position and Orientation of a Drill Pipe”, filed Sep.17, 2018, the entire disclosure of which is incorporated herein byreference.

FIELD OF THE DISCLOSURE

Embodiments of the current disclosure are directed toward rig drillfloor operations, and more particularly, apparatus, methods and systemsfor identifying a stick-up height, pose, and/or orientation of a drillpipe.

BACKGROUND

Drilling environments in the energy industry can be harsh on equipmentsuch as drill bits, pipes, tool joints, etc., requiring theirreplacement when the equipment are no longer able to functionadequately. When replacing distressed equipment, pipes are pulled outof, and put back into, wellbores in processes respectively known astripping out and tripping in. Tripping is generally viewed as anunproductive use of time and traditionally includes human intervention.Human intervention in some instances limits efficiency and introducesundesirable variability into the tripping process. Accordingly, there isa desire to automate the processes associated with tripping to reducethe inefficiencies and variability resulting from human intervention,including, for example, at least some operations of iron roughnecks thatare used to make or break the threaded joints of a drill string. In suchinstances, proper placement of the iron roughneck relative to the drillstring involves a very small margin of error, and so automating the ironroughneck allows for repeatable and predictable placement of the ironroughneck within that margin of error, thereby optimizing the trippingprocess. To ensure proper placement of the iron roughneck for trippingpurposes, for example, there is a need for accurate detection of thelocation and orientation of certain components of the drill string(e.g., the tool joint of a pipe extending upward from the slip).

SUMMARY

In some embodiments, a processor is operably coupled to a time of flight(TOF) camera, a light detection and ranging (LIDAR) sensor, and anoptical camera. The processor can receive from the TOF camera a TOFsignal representative of a first coordinate associated with a stick-upheight of a tool joint of a pipe of a drill string during a trippingoperation on a rig drill floor, a TOF signal representative of a pitchof the tool joint, and a TOF signal representative of a roll of the tooljoint. The processor can receive from the LIDAR sensor a LIDAR signalrepresentative of a second coordinate associated with the stick-upheight of the tool joint, and a LIDAR signal representative of the pitchof the tool joint. The processor can receive from the optical camera anoptical camera signal representative of a third coordinate associatedwith the stick-up height of the tool joint and an optical camera signalrepresentative of the roll of the tool joint. The processor can generatea pose estimate of the stick-up height of the tool joint based on thefirst coordinate, the second coordinate, and the third coordinate. Theprocessor can generate an orientation estimate of the tool joint basedon the TOF signal representative of the pitch of the tool joint, the TOFsignal representative of the roll of the tool joint, the LIDAR signalrepresentative of the pitch of the tool joint, and the optical camerasignal representative of the roll of the tool joint. The processor cansend an instruction signal, based on the pose estimate and theorientation estimate, such that a roughneck on the rig drill floor movesrelative to the tool joint.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts discussed in greater detail below (provided suchconcepts are not mutually inconsistent) are contemplated as being partof the inventive subject matter disclosed herein. In particular, allcombinations of claimed subject matter appearing at the end of thisdisclosure are contemplated as being part of the inventive subjectmatter disclosed herein. It should also be appreciated that terminologyexplicitly employed herein that also may appear in any disclosureincorporated by reference should be accorded a meaning most consistentwith the particular concepts disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The skilled artisan will understand that the drawings primarily are forillustrative purposes and are not intended to limit the scope of theinventive subject matter described herein. The drawings are notnecessarily to scale; in some instances, various aspects of theinventive subject matter disclosed herein may be shown exaggerated orenlarged in the drawings to facilitate an understanding of differentfeatures. In the drawings, like reference characters generally refer tolike features (e.g., functionally similar and/or structurally similarelements).

FIG. 1A is an example block diagram illustrating the characterization ofthe stick-up height and orientation of the tool-joint of a pipe during atrip-in or trip-out process so that accurate positions for placing thespinning wrench and the torque wrench of a roughneck on the pipe can beaccurately determined, according to some embodiments.

FIGS. 1B-E show various views of a pipe at a wellbore with non-zeropitch and/or roll characterizing an orientation of the pipe (and itsassociated tool joint), according to some embodiments.

FIG. 2 shows a flowchart illustrating the calculation of the stick-upheight and orientation of the tool-joint of a pipe during a trip-in ortrip-out process so that accurate positions for placing the spinningwrench and the torque wrench of a roughneck on the pipe can beaccurately determined, according to some embodiments.

FIG. 3A illustrates a drill string having a portion extending from awell bore, and FIG. 3B illustrates an example tool joint, according tosome embodiments.

FIGS. 4A-4D show example images of an experimental demonstration of thecharacterization of the stick-up height and orientation of a drill pipebased on the embodiments disclosed herein, according to someembodiments.

DETAILED DESCRIPTION

In some embodiments, a drill string can be used to drill wellbores,which are the holes that are formed in the Earth's sub-surface tofacilitate the extraction of natural resources such as oil, gas,minerals, water, etc. A drill string may include a drill bit for cuttinginto the earth and a plurality of drill pipes connected at the tooljoints of the pipes. To replace any worn out part of the drill string,in some embodiments, the drill string may be pulled out of the ground(“tripping out” process) or put back into the ground (“tripping in”process) by disconnecting or attaching, respectively, at least some tooljoints of the pipes of the drill strings. For example, when replacing adrill bit, the drill string may be tripped out by pulling the string outof the wellbore while disconnecting some or all of the pipe connectionpoints at the tool joints. Upon replacing the drill bit, in someembodiments, the drill string can be put back into the wellbore whileattaching drill pipe(s) onto the drill string. It is to be noted thatthe tripping process is not limited to replacing drill bits, but canalso be employed to replace other parts of the drill string such as thepipes themselves.

In some embodiments, an iron roughneck that includes a spinning wrenchand a torque wrench may be used to attach or disconnect pipes at thetool-joints. For example, during a tripping out process, a drill stringmay be pulled out of a wellbore exposing a pipe connection point at thetool joints of the pipes and the torque wrench of a roughneck may beused to stabilize the drill string by locking onto the drill stringbelow the pipe connection point while the spinning wrench locks onto thepipe above the pipe connection point and disconnects the threadedconnection point by rotating the pipe relative to the drill string.During the tripping in process, the reverse may take place, a pipe beingadded to the drill string with the spinning wrench, locked onto thepipe, turning the pipe relative to the drill string to tighten the pipeonto the drill string at the tool joint, which then can be followed bythe drill string being lowered into the wellbore.

In some embodiments, automating the proper placement of the ironroughneck (alternatively the spinning wrench and/or the torque wrench)on the drill string and/or pipes during a tripping in or tripping outprocess may enhance efficiency, amongst other things, and as such may bedesirable. For example, the positions of the spinning wrench and/or thetorque wrench within the roughneck may be adjusted so that when theroughneck makes contact with a drill string to form a pipe connection ordisconnect it, the wrenches are positioned at least substantiallyaccurately on the pipes adjacent to the pipe connection. In someembodiments, the at least substantially accurate positioning of thewrenches on the pipes may be facilitated by at least substantiallyaccurate determination of the stick-up height of the drill string, i.e.,the height of the drill string sticking out of the wellbore. In someembodiments, the stick-up height may be defined as the height to the topof the tool joint of the pipe that is a part of the drill string andsticking out of the wellbore. Further, in some embodiments, theorientation of the tool joint can also be used to improve upon thedetermination of the stick-up height and/or to facilitate the at leastsubstantially accurate positioning of the wrenches on the pipes.

FIG. 1A presents an example block diagram illustrating thecharacterization of the stick-up height and orientation of thetool-joint of a pipe during a trip-in or trip-out process so that atleast substantially accurate positions for placing the spinning wrenchand/or the torque wrench of a roughneck on the pipe can be accuratelydetermined, according to some embodiments. In some embodiments, a drillstring 108 may be tripped in (i.e., put into) or tripped out (i.e.,pulled out of) a wellbore 126 that includes a slip 106 at its mouth forsuspending the drill string 108 in the wellbore 126. In someembodiments, during tripping in, an additional pipe 124 is beingattached to the top pipe 102 of the drill string 108 (e.g., beingattached to the tool joint 104 of the pipe 102 of the drill string 108)before the drill string is lowered into the wellbore 126, while duringtripping out, the pipe 124 is being removed from the drill string 108(e.g., disconnected from the tool joint 104 of the top pipe 102 of thedrill string 108) before the drill string 108 is further pulled out toexpose additional pipes or pipe joint tools. For example, the tool joint104 may be threaded, and the attachment or detachment of the pipe 124from the top pipe 102 may occur by screwing the pipe 124 onto or out ofthe tool joint 104 using the spinning wrench 120 and the torque wrench122 of the roughneck 110.

As discussed above, in some embodiments, efficient attachment ordetachment of the pipe 124 from the top pipe 102, including the use ofsubstantially or fully automated techniques, may be facilitated by theat least substantially accurate determination of the stick-up height 128and/or the orientation of the tool joint 104. In some embodiments, oneor more sensing technologies, individually and/or in combination, may beutilized to at least substantially accurately determine or estimate thestick-up height 128 and/or the orientation of the tool joint 104. Theuse of a combination of a plurality of technologies may also aid withovercoming external limitations such as environmental effects as sometechnologies are less prone to such effects and the combined resultwould be robust against the limitations. In some embodiments, thestick-up height 128 can be the height of the part of the pipe jutting orsticking out of the wellbore 126 or drill floor. For example, thestick-up height can be understood as the height of the part of the pipe102 sticking out from the drill floor or the mouth of the wellbore 126to some selected position on the tool joint 104 of the same pipe 102;examples of said selected position including the top, bottom, middle,etc., of the tool joint 104.

In some embodiments, the orientation of the tool joint 104 (oralternatively the pipe 102) may be characterized by one or more of thepitch, the roll and/or the yaw of the pipe 102. The roll, the pitch andthe yaw can be defined as rotations about a three-dimensional (3D) bodysuch as the pipe 102 about the three orthogonal axes defining the 3Dbody (e.g., the x-axis, the y-axis and the z-axis, respectively). Forexample, with reference to FIGS. 1B-E, the pipe 102 is shown to beoffset from its longitudinal axis, indicating non-zero values for one orboth of the pitch and the roll. FIGS. 1C-E, showing bird's-eye views ofthe x-y plane view of the pipe 102, illustrate a combination of a pitchand a roll, a pitch, and a roll, respectively.

In some embodiments, one or more of light detection and ranging (LIDAR)sensor 114, time of flight (TOF) camera/sensor 112 and optical camera(OC) 116 can be used to generate data that is then provided to aprocessor 118 for processing to determine or estimate the stick-upheight 128 and/or the orientation of the tool joint 104. Upon theprocessor 118 determining or estimating the stick-up height 128 and/orthe orientation of the tool joint 104, in some embodiments, theprocessor may then generate instructions to the roughneck 110 to adjustthe positions of one or all of the spinning wrench 120, the torquewrench 122 and the roughneck 110 itself on the pipe 102 and/or the pipe124 such that the attachment or detachment of pipe 124 to or from pipe102 may occur efficiently and safely. For example, the attachment ordetachment of pipe 124 to or from pipe 102 may occur in an automatedfashion without human intervention, i.e., without a human adjusting thepositioning of the roughneck 110 (or the spinning wrench 120 and/or thetorque wrench 122). Further, with a sufficiently accurate estimate ofthe stick-up height and orientation of the pipe(s) 102, 124, an area oramount of surface contact between the roughneck 110 and the pipe(s) 102,124 can be maximized and/or optimized such that transfer of force and/ortorque from the roughneck 110 to the pipe(s) 102, 124 can be optimized.

In some embodiments, the LIDAR sensor 114 can be used to scan the pipe102, and in particular the tool joint 104, with pulsed light waves 130 aand receive the reflected pulses 130 b, which allow for thedetermination of the profile of the pipe stand jutting out of the mouthof the wellbore 126 past the slip 106 (e.g., a 1D point cloud of theobjects illuminated with the pulsed light waves, a one-dimensional(“1D”) point cloud being a collection of points representing the 1Dcoordinates of the objects (and hence outlining shapes or features ofthe objects)). For example, the pulsed light waves 130 a may be a laser,including laser having wavelength in the range from about 500 nm toabout 1600 nm, wavelengths of about 532 nm, about 905 nm, about 1064 nm,about 1550 nm, etc., including values and subranges therebetween. Insome embodiments, since the LIDAR sensor 114 produces its own lightsource, it may be at least substantially unaffected by variable lightingscenarios and may require little or no external light source.

In some embodiments, an analysis of the 1D point cloud collected by theLIDAR 114 provides the stick-up height 128 and the pitch, if any, of thetool joint 104 of the pipe 102 (e.g., the top-end portion of the tooljoint 104). For example, with the LIDAR 114 directed at or near the pipe102, reflected waves received by the LIDAR 114 within a certain timethreshold or time range after transmission may be deemed to have beenreflected by the pipe 102, while the rest may be considered to have beenreflected by other objects in the surroundings. Further to this example,the LIDAR 114 can scan the pipe 102 in an upward direction along theZ-axis (i.e., in the vertical direction), and in turn the LIDAR 114 willreceive and/or sense waves reflected from the pipe 102 until the LIDAReventually scans above and/or beyond the pipe 102, in which case theLIDAR will receive and/or sense waves reflected from an object otherthan the pipe 102 or the LIDAR will not receive and/or sense anyreflected waves (e.g., if there are no objects beyond or behind the pipe102 that are within the LIDAR's range). The time period between theLIDAR's light transmission and receipt of one or more waves reflectedfrom the pipe 102 will be different from the time period between theLIDAR's light transmission and receipt of one or more waves reflectedfrom an object other than the pipe 102 (and/or not receiving reflectedwaves at all). This difference in time can be used to identify thestick-up height 128 of the pipe 102. As such, by analyzing the 1D pointcloud obtained as the LIDAR 114 is scanning along the height of thepipe, in some embodiments, data from the LIDAR 114 can be used toidentify which data points belong to reflections from the pipe 102 andextract or calculate the stick-up height 128 of the tool joint 104 fromthe profile of the identified data points. Further, data from the LIDAR114 may also be used to detect and determine or estimate a pitch of thepipe 102 from the same identified data points, as the arrival times ofthe reflected waves would be different than what would be expected ifthe pipe 102 was not tilted (i.e., a pitch in the pipe 102 would resultin changes in arrival times of the reflected waves that reflect fromcontact with the pipe). For example, an arrival time for a reflectionfrom a first transmitted wave that is greater than an arrival time for areflection from a second transmitted wave as the LIDAR scans upward mayindicate that the pipe 102 is pitched towards the LIDAR 114.Accordingly, an analysis of the 1D point cloud can provide the stick-upheight 128 and pitch of the tool joint 104, facilitating the at leastsubstantially accurate determination of the top of the tool joint 104that may be configured to receive another pipe 124 during tripping in,or from which the pipe 124 can be detached or removed during trippingout.

In some embodiments, a TOF camera or sensor 112 can also be used toobtain height and/or orientation data related to the tool joint 104. ATOF sensor 112 may be a scannerless device that can illuminate an entirescene of interest (e.g., the tool joint 104 as well as the pipe 102, theslip 106, and/or the pipe 124 if present and of interest, etc.) withlight emitted from a laser source in the TOF sensor 112, and thenreceive the reflected light for processing to determine and/or estimatethe stick-up height 128 and/or the orientation of the tool joint 104. Insome embodiments, and similar to a LIDAR detector, the TOF sensor 112receives and measures reflections of its own emitted light; as such, theTOF sensor 112 is at least substantially immune to effects of externallight sources such as ambient light. Based on the reflected light, andin particular based on phase and/or travel time information in thereflected light when compared to the transmitted light, in someembodiments, the TOF sensor 112 can extract a three-dimensional (“3D”)point cloud of the scene of interest, which allows for the at leastsubstantially accurate determination and/or estimation of one or both ofthe height (e.g., stick-up height) and/or orientation of the tool joint104. For example, the 3D point cloud allows for the extraction of thestick up height 128 as well as two degrees of freedom in the orientationof the pipe 102/the tool joint 104, the pitch and the roll. Based on theextracted stick-up height 128, the pitch and/or the roll, in someembodiments, the TOF sensor 112 can at least substantially accuratelydetermine the location of the top of the tool joint 104 that may beconfigured to receive another pipe 124 during tripping in, or from whichthe pipe 124 can be detached or removed during tripping out.

In some embodiments, an optical camera (“OC”) 116 may also be used toobtain an image of the tool joint 104 (and its surroundings includingthe pipe 102), and convolutional neural networks (CNNs) may be appliedto the two-dimensional (“2D”) image to extract or recover a 3Dreconstruction of the tool joint 104 and the pipe 102, thereby allowingfor an at least substantially accurate determination or estimate of itsheight and/or orientation. In some embodiments, the CNNs may be trainedto recognize the pipe 102 (and its part the tool joint 104) from imagesobtained by the OC 116 by training the CNNs with a variety of pipeand/or tool joint 104 types (e.g., shapes, features, color, etc.), pipeand/or tool joint 104 sizes (e.g., diameters, lengths, etc.), weatherconditions (of the location of the wellbore 126), ambient lightconditions, etc. With the CNN's learning-based recognition of the pipe102 and the tool joint 104, in some embodiments, the stick-up height 128and the orientation of the tool joint 104 can be determined orestimated. For example, the CNN can be trained to determine or estimatethe stick-up height 128 and/or the roll of the tool joint 104 based onimages obtained by the OC 116, allowing for the at least substantiallyaccurate determination of the top of the tool joint 104 that may beconfigured to receive another pipe 124 during tripping in, or from whichthe pipe 124 can be detached or removed during tripping out.

As discussed above, data on the stick-up height 128 and/or orientation(e.g., pitch, roll) of the tool joint 104 may be obtained from a varietyof methods or techniques, i.e., using one or more of the LIDAR sensor114, the TOF camera 112 and/or the OC 116. In some embodiments, the dataobtained from these sources may be combined to extract more accuratevalues for the stick-up height and/or the orientation of the tool joint104, and as a consequence a more accurate determination for the locationof the top of the tool joint 104. For example, the processor 118 mayreceive these data from one or more of the LIDAR sensor 114, the TOFcamera 112 and/or the OC 116, and proceed to calculate the averages foreach quantity (i.e., stick-up height, pitch and roll) to obtain theenhanced values. In some embodiments, the results obtained from somemethods, techniques or device types can be given more weight incomparison to others based on, for example, ambient and/or environmentalconditions. For example, if the data were taken during poor weather orlight conditions, the data from the LIDAR sensor 114 and the TOF camera112 may be given more weight in comparison to data from the OC 116. Insome embodiments, the use of a variety of techniques and methods todetermine the stick-up height and the orientation of the tool joint 104allows for the results to be robust against harsh environmentalconditions such as darkness, rain, etc.

FIG. 2 shows an example flowchart illustrating the determination of thestick-up height, pose, and orientation of the pipe as discussed above,according to some embodiments. As shown, a system may include a poseestimator to generate a pose estimate of a drill pipe based on Zestimates of the stick-up height of the drill pipe based on signalsgenerated by the TOF sensor, the LIDAR sensor, and the optical camera.Further, the system may include an orientation estimator to generate anorientation estimate of the drill pipe based on a pitch and rollestimate generated from the TOF sensor, a pitch estimate generated fromthe LIDAR sensor, and a roll estimate generated from the optical camera,as described in further detail herein.

Further to the discussion herein with respect to the 3D point clouddata, in some embodiments, when evaluating the point cloud data,estimation of pitch and roll values can be calculated by associatingand/or fitting cylinders to the point cloud data. In turn, orientationcan be obtained based at least in part on the cylinders discovered. The1D point cloud, on the other hand, provides data in a single plane. Thesensor (e.g., the LIDAR sensor) can be positioned in a way that the 1Dpoint cloud data intersects the pitch plane. Fitting a line through thedata will thereby inherently give the pitch value. Further, similar toas discussed in more detail here, the optical data provides the abilityto determine the roll estimate. To this end, a line detection algorithmcan be defined and/or used to identify and/or find the edges of thepipe. The angle of the line in reference to the ground plane can be usedto identify the roll value.

Referring back to FIG. 1 , upon the processor 118 determining orestimating an enhanced or more accurate value for the stick-up height128 and/or the orientation of the tool joint 104, in some embodiments,the processor may then generate and/or send instructions to theroughneck 110 to adjust the positioning of one or all of the spinningwrench 120, the torque wrench 122 and the roughneck 110 itself on thepipe 102, the tool joint 104 and/or the pipe 124 (e.g., in connectionwith a tripping in or tripping out operation).

In some embodiments, a combination of two or more of the LIDAR sensor114, the TOF camera 112 or the OC 116 can be used to determine thelocation of the seam between pipes (e.g., between the tool joints of thepipes) during a tripping out process. For example, an accuratedetermination of the location of the seam allows the roughneck to bedirected to and/or placed on the pipes accurately and in an automatedfashion such that detaching or separating of the pipes can occurroutinely and in a repeatable manner without human intervention. Withreference to FIG. 3A, a drill string 310 and wellbore 314 are shown inwhich pipes 302, 316 are connected to each other above the wellbore 314and via their respective tool joints 304, 308. FIG. 3B illustrates anexample of tool joint 304. In some embodiments, identifying the locationof the seam 312, i.e., where the pipes 302, 316 meet, would be useful indetermining where on the pipes the roughneck should make contact toseparate the pipes 302, 316. In such cases, a LIDAR sensor can be usedto scan the pipes 302, 316, including particular identifying portions ofthe pipes 302, 316, such as the shoulders 320, 322, using pulsed lightwaves to generate a 1D point cloud of the shoulders 320, 322 (which canbe used to identify the transitions to the tool joints 304, 308 from theremainder of the pipes 302, 316). The 1D point cloud provides theprofile of the pipes 302, 316, which includes the transitions of theshoulders 320, 322 into the tool joints 304, 308 before terminating atthe seam where the two pipes 302, 316 meet. As such, by analyzing the 1Dpoint cloud, in some embodiments, a region of interest (ROI) 306encompassing the shoulders 320 and 322, and including the seam 312therein, can be identified. In some embodiments, a TOF sensor can alsobe used to generate a 3D point cloud, from which the ROI 306 can beidentified. In some embodiments, a CNN may also be utilized to analyzeimages of the pipes 302, 316, and in particular images of the shoulders320,322, and/or any other suitable landmark (such as a fiducial attachedto and/or apart of the pipe(s)) to identify the ROI 306. In someembodiments, by combining the results of the LIDAR sensor, the TOFsensor and/or the CNN, an average and/or calculated (e.g., weightedcalculation) ROI 306 that includes the seam 312 may be defined.

In some embodiments, upon the identification of the ROI, the CNN mayprocess images of the ROI to determine the location of the seam 312within the ROI and identify one or more possible candidate locations forthe seam 312. Further, in some embodiments, the LIDAR sensor and the TOFsensor may also be used to determine the location of the seam 312 and/orobtain a more accurate location of it (e.g., if the CNN does not returna valid candidate or too many candidates are identified). In someembodiments, the location of the seam 312 may be obtained with a certainconfidence or probability level. In some embodiments, the probabilitylevel may not be 100% (e.g., when the edge of the tool joint of a pipeis beveled and visibility is poor). In such embodiments, a threshold(e.g., user-defined threshold) can be used to determine if theprobability level is high enough, and if so, the obtained location ofthe seam 312 can be deemed accurate and/or sufficient for directing theroughneck. In some embodiments, and in particular if the CNN fails tolocate the seam (e.g., if the above probability level falls below thethreshold), the equidistant point between the two tool joints 304, 308may be considered as the seal 312 location.

FIGS. 4A-D show example images of an experimental demonstration of thecharacterization of the stick-up height and orientation of a drill pipebased on the embodiments disclosed herein, according to someembodiments. FIGS. 4B and 4D show cloud points obtained from LIDAR andTOF sensor measurements of the drill pipe shown in the optical cameraimages in FIGS. 4A and 4C, respectively. The locations of the top of thepipe as determined by the LIDAR and the TOF sensors are fairly close tothe actual results, illustrating the applicability of the systems,apparatus and methods disclosed herein for characterizing stick-upheights and orientations of drill pipes.

While various inventive embodiments have been described and illustratedherein, those of ordinary skill in the art will readily envision avariety of other means and/or structures for performing the functionand/or obtaining the results and/or one or more of the advantagesdescribed herein, and each of such variations and/or modifications isdeemed to be within the scope of the inventive embodiments describedherein. More generally, those skilled in the art will readily appreciatethat all parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the inventive teachingsis/are used. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presentedby way of example only and that, within the scope of the appended claimsand equivalents thereto; inventive embodiments may be practicedotherwise than as specifically described and claimed. Inventiveembodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the inventive scope of the present disclosure.

The above-described embodiments can be implemented in any of numerousways. For example, embodiments of the present technology may beimplemented using hardware, firmware, software or a combination thereof.When implemented in firmware and/or software, the firmware and/orsoftware code can be executed on any suitable processor or collection oflogic components, whether provided in a single device or distributedamong multiple devices.

In this respect, various inventive concepts may be embodied as acomputer readable storage medium (or multiple computer readable storagemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other non-transitory medium or tangible computer storagemedium) encoded with one or more programs that, when executed on one ormore computers or other processors, perform methods that implement thevarious embodiments of the invention discussed above. The computerreadable medium or media can be transportable, such that the program orprograms stored thereon can be loaded onto one or more differentcomputers or other processors to implement various aspects of thepresent invention as discussed above.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of embodiments as discussedabove. Additionally, it should be appreciated that according to oneaspect, one or more computer programs that when executed perform methodsof the present invention need not reside on a single computer orprocessor, but may be distributed in a modular fashion amongst a numberof different computers or processors to implement various aspects of thepresent invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Also, various inventive concepts may be embodied as one or more methods,of which an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

The invention claimed is:
 1. An apparatus, comprising a processoroperably coupled to a time of flight (TOF) camera, a light detection andranging (LIDAR) sensor, and an optical camera, the processor configuredto receive, from the TOF camera, (1) a TOF signal representative of afirst coordinate associated with a stick-up height of a tool joint of apipe of a drill string during a tripping operation on a rig drill floor,(2) a TOF signal representative of a pitch of the tool joint, and (3) aTOF signal representative of a roll of the tool joint, the processorconfigured to receive, from the LIDAR sensor, (1) a LIDAR signalrepresentative of a second coordinate associated with the stick-upheight of the tool joint, and (2) a LIDAR signal representative of thepitch of the tool joint, the processor configured to receive, from theoptical camera, (1) an optical camera signal representative of a thirdcoordinate associated with the stick-up height of the tool joint, and(2) an optical camera signal representative of the roll of the tooljoint, the processor configured to generate a pose estimate of thestick-up height of the tool joint based on the first coordinate, thesecond coordinate, and the third coordinate, the processor configured togenerate an orientation estimate of the tool joint based on the TOFsignal representative of the pitch of the tool joint, the TOF signalrepresentative of the roll of the tool joint, the LIDAR signalrepresentative of the pitch of the tool joint, and the optical camerasignal representative of the roll of the tool joint, the processorconfigured to send an instruction signal such that a roughneck on therig drill floor moves relative to the tool joint based on the poseestimate and the orientation estimate.
 2. The apparatus of claim 1,wherein: the TOF signal representative of the first coordinate, the TOFsignal representative of the pitch of the tool joint, and the TOF signalrepresentative of the roll of the tool joint are generated based atleast in part on a three-dimensional point cloud.
 3. The apparatus ofclaim 1, wherein: the LIDAR signal representative of the secondcoordinate and the LIDAR signal representative of the pitch of the tooljoint are based at least in part on a one-dimensional point cloud. 4.The apparatus of claim 1, wherein: the optical camera signalrepresentative of the third coordinate and the optical camera signalrepresentative of the roll of the tool joint are based at least in parton two-dimensional image data.
 5. The apparatus of claim 1, wherein: theprocessor is configured to send the instruction signal such that theroughneck on the rig drill floor moves towards and into operableengagement with the pipe of the drill string.
 6. A method, comprising:receiving, at a processor and from a time of flight (TOF) camera, (1) aTOF signal representation of a first coordinate associated with astick-up height of a tool joint of a pipe of a drill string during atripping operation on a rig drill floor, (2) a TOF signal representativeof a pitch of the tool joint, and (3) a TOF signal representative of aroll of the tool joint; receiving, at the processor and from a lightdetection and ranging (LIDAR) sensor, (1) a LIDAR signal representativeof a second coordinate associated with the stick-up height of the tooljoint, and (2) a LIDAR signal representative of the pitch of the tooljoint; receiving, at the processor and from an optical camera, (1) anoptical camera signal representative of a third coordinate associatedwith the stick-up height of the tool joint, and (2) an optical camerasignal representative of the roll of the tool joint; generating, at theprocessor, a pose estimate of the stick-up height of the tool jointbased on the first coordinate, the second coordinate, and the thirdcoordinate; generating, at the processor, an orientation estimate of thetool joint based on the TOF signal representative of the pitch of thetool joint, the TOF signal representative of the roll of the tool joint,the LIDAR signal representative of the pitch of the tool joint, and theoptical camera signal representative of the roll of the tool joint; andsending, from the processor, an instruction signal such that a roughneckon the rig drill floor moves relative to the tool joint based on thepose estimate and the orientation estimate.
 7. The method of claim 6,wherein: the TOF signal representative of the first coordinate, the TOFsignal representative of the pitch of the tool joint, and the TOF signalrepresentative of the roll of the tool joint are generated based atleast in part on a three-dimensional point cloud.
 8. The method of claim6, wherein: the LIDAR signal representative of the second coordinate andthe LIDAR signal representative of the pitch of the tool joint are basedat least in part on a one-dimensional point cloud.
 9. The method ofclaim 6, wherein: the optical camera signal representative of the thirdcoordinate and the optical camera signal representative of the roll ofthe tool joint are based at least in part on two-dimensional image data.10. The method of claim 6, wherein: the sending the instruction signalincludes sending the instruction signal such that the roughneck on therig drill floor moves towards and into operable engagement with the pipeof the drill string.